<!DOCTYPE html>


<html lang="zh-CN">


<head>
  <meta charset="utf-8" />
    
  <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1" />
  <title>
    Kubernetes 新玩法：在 yaml 中编程.md |  
  </title>
  <meta name="generator" content="hexo-theme-ayer">
  
  <link rel="shortcut icon" href="/favicon.ico" />
  
  
<link rel="stylesheet" href="/dist/main.css">

  
<link rel="stylesheet" href="https://cdn.jsdelivr.net/gh/Shen-Yu/cdn/css/remixicon.min.css">

  
<link rel="stylesheet" href="/css/custom.css">

  
  
<script src="https://cdn.jsdelivr.net/npm/pace-js@1.0.2/pace.min.js"></script>

  
  

  

</head>

</html>

<body>
  <div id="app">
    
      
    <main class="content on">
      <section class="outer">
  <article
  id="post-k8s/Kubernetes 新玩法：在 yaml 中编程"
  class="article article-type-post"
  itemscope
  itemprop="blogPost"
  data-scroll-reveal
>
  <div class="article-inner">
    
    <header class="article-header">
       
<h1 class="article-title sea-center" style="border-left:0" itemprop="name">
  Kubernetes 新玩法：在 yaml 中编程.md
</h1>
 

    </header>
     
    <div class="article-meta">
      <a href="/2020/11/11/k8s/Kubernetes%20%E6%96%B0%E7%8E%A9%E6%B3%95%EF%BC%9A%E5%9C%A8%20yaml%20%E4%B8%AD%E7%BC%96%E7%A8%8B/" class="article-date">
  <time datetime="2020-11-10T16:00:00.000Z" itemprop="datePublished">2020-11-11</time>
</a> 
  <div class="article-category">
    <a class="article-category-link" href="/categories/k8s/">k8s</a>
  </div>
  
<div class="word_count">
    <span class="post-time">
        <span class="post-meta-item-icon">
            <i class="ri-quill-pen-line"></i>
            <span class="post-meta-item-text"> 字数统计:</span>
            <span class="post-count">2.8k</span>
        </span>
    </span>

    <span class="post-time">
        &nbsp; | &nbsp;
        <span class="post-meta-item-icon">
            <i class="ri-book-open-line"></i>
            <span class="post-meta-item-text"> 阅读时长≈</span>
            <span class="post-count">13 分钟</span>
        </span>
    </span>
</div>
 
    </div>
      
    <div class="tocbot"></div>




  
    <div class="article-entry" itemprop="articleBody">
       
  <h1 id="Kubernetes-新玩法：在-yaml-中编程"><a href="#Kubernetes-新玩法：在-yaml-中编程" class="headerlink" title="Kubernetes 新玩法：在 yaml 中编程"></a>Kubernetes 新玩法：在 yaml 中编程</h1><p><strong>引子</strong></p>
<p>性能测试在日常的开发工作中是常规需求，用来摸底服务的性能。</p>
<p>那么如何做性能测试？要么是通过编码的方式完成，写一堆脚本，用完即弃；要么是基于平台，在平台定义的流程中进行。对于后者，通常由于目标场景的复杂性，如部署特定的 workload、观测特定的性能项、网络访问问题等，往往导致性能测试平台要以高成本才能满足不断变化的开发场景的需求。</p>
<p>在云原生的背景下，是否可以更好解决这种问题？</p>
<p>先看两个 yaml 文件：</p>
<ul>
<li>performance-test.yaml 描述了在 K8s 中的操作流程：</li>
</ul>
<ol>
<li><ol>
<li>创建测试用的 Namespace</li>
<li>启动针对 Deployment 创建效率和创建成功率的监控</li>
<li>下述动作重复 N 次：① 使用 workload 模板创建 Deployment；② 等待 Deployment 变为 Ready</li>
<li>删除测试用的 Namespace</li>
</ol>
</li>
<li></li>
</ol>
<ul>
<li>basic-1-pod-deployment.yaml 描述使用的 workload 模板</li>
</ul>
<p>performance-test.yaml ：</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br></pre></td><td class="code"><pre><span class="line"></span><br><span class="line">apiVersion: aliyun.com&#x2F;v1alpha1</span><br><span class="line">kind: Beidou</span><br><span class="line">metadata:</span><br><span class="line">  name: performance</span><br><span class="line">  namespace: beidou</span><br><span class="line">spec:</span><br><span class="line">  steps:</span><br><span class="line">  - name: &quot;Create Namespace If Not Exits&quot;</span><br><span class="line">    operations:</span><br><span class="line">    - name: &quot;create namespace&quot;</span><br><span class="line">      type: Task</span><br><span class="line">      op: CreateNamespace</span><br><span class="line">      args:</span><br><span class="line">      - name: NS</span><br><span class="line">        value: beidou</span><br><span class="line">  - name: &quot;Monitor Deployment Creation Efficiency&quot;</span><br><span class="line">    operations:</span><br><span class="line">    - name: &quot;Begin To Monitor Deployment Creation Efficiency&quot;</span><br><span class="line">      type: Task</span><br><span class="line">      op: DeploymentCreationEfficiency</span><br><span class="line">      args:</span><br><span class="line">      - name: NS</span><br><span class="line">        value: beidou</span><br><span class="line">    - name: &quot;Repeat 1 Times&quot;</span><br><span class="line">      type: Task</span><br><span class="line">      op: RepeatNTimes</span><br><span class="line">      args:</span><br><span class="line">      - name: TIMES</span><br><span class="line">        value: &quot;1&quot;</span><br><span class="line">      - name: ACTION</span><br><span class="line">        reference:</span><br><span class="line">          id: deployment-operation</span><br><span class="line">  - name: &quot;Delete namespace&quot;</span><br><span class="line">    operations:</span><br><span class="line">    - name: &quot;delete namespace&quot;</span><br><span class="line">      type: Task</span><br><span class="line">      op: DeleteNamespace</span><br><span class="line">      args:</span><br><span class="line">      - name: NS</span><br><span class="line">        value: beidou</span><br><span class="line">      - name: FORCE</span><br><span class="line">        value: &quot;false&quot;</span><br><span class="line">  references:</span><br><span class="line">  - id: deployment-operation</span><br><span class="line">    steps:</span><br><span class="line">    - name: &quot;Prepare Deployment&quot;</span><br><span class="line">      operations:</span><br><span class="line">      - name: &quot;Prepare Deployment&quot;</span><br><span class="line">        type: Task</span><br><span class="line">        op: PrepareBatchDeployments</span><br><span class="line">        args:</span><br><span class="line">        - name: NS</span><br><span class="line">          value: beidou</span><br><span class="line">        - name: NODE_TYPE</span><br><span class="line">          value: ebm</span><br><span class="line">        - name: BATCH_NUM</span><br><span class="line">          value: &quot;1&quot;</span><br><span class="line">        - name: TEMPLATE</span><br><span class="line">          value: &quot;.&#x2F;templates&#x2F;basic-1-pod-deployment.yaml&quot;</span><br><span class="line">        - name: DEPLOYMENT_REPLICAS</span><br><span class="line">          value: &quot;1&quot;</span><br><span class="line">        - name: DEPLOYMENT_PREFIX</span><br><span class="line">          value: &quot;ebm&quot;</span><br><span class="line">      - name: &quot;Wait For Deployments To Be Ready&quot;</span><br><span class="line">        type: Task</span><br><span class="line">        op: WaitForBatchDeploymentsReady</span><br><span class="line">        args:</span><br><span class="line">        - name: NS</span><br><span class="line">          value: beidou</span><br><span class="line">        - name: TIMEOUT</span><br><span class="line">          value: &quot;3m&quot;</span><br><span class="line">        - name: CHECK_INTERVAL</span><br><span class="line">          value: &quot;2s&quot;</span><br></pre></td></tr></table></figure>

<p>basic-1-pod-deployment.yaml：</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br></pre></td><td class="code"><pre><span class="line"></span><br><span class="line">apiVersion: apps&#x2F;v1</span><br><span class="line">kind: Deployment</span><br><span class="line">metadata:</span><br><span class="line">  labels:</span><br><span class="line">    app: basic-1-pod</span><br><span class="line">spec:</span><br><span class="line">  selector:</span><br><span class="line">    matchLabels:</span><br><span class="line">      app: basic-1-pod</span><br><span class="line">  template:</span><br><span class="line">    metadata:</span><br><span class="line">      labels:</span><br><span class="line">        app: basic-1-pod</span><br><span class="line">    spec:</span><br><span class="line">      containers:</span><br><span class="line">      - name: nginx</span><br><span class="line">        image: registry-vpc.cn-hangzhou.aliyuncs.com&#x2F;xxx&#x2F;nginx:1.17.9</span><br><span class="line">        imagePullPolicy: Always</span><br><span class="line">        resources:</span><br><span class="line">          limits:</span><br><span class="line">            cpu: 2</span><br><span class="line">            memory: 4Gi</span><br></pre></td></tr></table></figure>

<p>然后通过一个命令行工具执行 performance-test.yaml：</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">$ beidou server -c ~&#x2F;.kube&#x2F;config services&#x2F;performance-test.yaml</span><br></pre></td></tr></table></figure>

<p>执行效果如下 (每个 Deployment 创建耗时，所有 Deployment 创建耗时的 TP95 值，每个 Deployment 是否创建成功)：</p>
<p><img src="http://iubest.gitee.io/pic/640-1601016578135.webp" alt="img"></p>
<p>这些 metrics 是按照 Prometheus 标准输出，可以被 Prometheus server 收集走，再结合 Grafana 可以可视化展示性能测试数据。</p>
<p>通过在 yaml 中表达想法，编排对 K8s 资源的操作、监控，再也不用为性能测试的实现头疼了 :D</p>
<h2 id=""><a href="#" class="headerlink" title=""></a></h2><p><img src="http://iubest.gitee.io/pic/640-1601016578097.gif" alt="动态黑色音符"></p>
<p><strong>为什么要在 yaml 中编程？</strong></p>
<p>性能测试、回归测试等对于服务质量保障有很大帮助，需要做，但常规的实现方法在初期需要投入较多的时间和精力，新增变更后维护成本比较高。</p>
<p>通常这个过程是以代码的方式实现原子操作，如创建 Deployment、检测 Pod 配置等，然后再组合原子操作来满足需求，如 创建 Deployment -&gt; 等待 Deployment ready -&gt; 检测 Pod 配置等。</p>
<p>有没有办法在实现的过程中既可以尽量低成本实现，又可以复用已有的经验？</p>
<p>可以将原子操作封装为原语，如 CreateDeployment、CheckPod，再通过 yaml 的结构表达流程，那么就可以通过 yaml 而非代码的方式描述想法，又可以复用他人已经写好的 yaml 文件来解决某类场景的需求。</p>
<p>即在 yaml 中编程，减少重复性代码工作，通过 <strong>声明式</strong> 的方式描述逻辑，并以 yaml 文件来满足场景级别的复用。</p>
<p>业界有很多种类型的 <strong>声明式操作</strong> 服务，如运维领域中的<a target="_blank" rel="noopener" href="https://mp.weixin.qq.com/s?__biz=MzUzNzYxNjAzMg==&mid=2247494380&idx=1&sn=8cac110d7f50d5a3852a8f69796db963&chksm=fae6e123cd9168359b9bba26f708d2b4013cfed0d592dd1d56daece5040b10367f2f51941a41&mpshare=1&scene=24&srcid=09189RwfTmxbBxMcfb2mLjxz&sharer_sharetime=1600403164443&sharer_shareid=407c90840c4caeeaf9680b1dd38c62ba&key=2ff6efa5b41f0127c9e924be5ce6384145d0c7b612a7633330d08c5fa39fbbfb2ab6c7d4b937423b54f24df15180312494985a7d0d2daa7a62627643cc000fa6dd852a5b9448a9f4044d0c688e5376586d1eef888fecca2dadd2ed1ca94685cce2afb8aa23ee2d4b122ab673e1209256add0569914cbc144befe150d14f9956d&ascene=14&uin=MTIwMjI3NTkwNQ==&devicetype=Windows+10+x64&version=62090529&lang=zh_CN&exportkey=AzxByEV7vLvuybMOAm6iNm4=&pass_ticket=pdKql0fF0rGOXvNr/tToA1+AardNoo77GWcTcNS7PpaVOYI2W/vk8qbSO4P5qmER&wx_header=0&winzoom=1">Ansible</a>、<a target="_blank" rel="noopener" href="https://mp.weixin.qq.com/s?__biz=MzUzNzYxNjAzMg==&mid=2247494380&idx=1&sn=8cac110d7f50d5a3852a8f69796db963&chksm=fae6e123cd9168359b9bba26f708d2b4013cfed0d592dd1d56daece5040b10367f2f51941a41&mpshare=1&scene=24&srcid=09189RwfTmxbBxMcfb2mLjxz&sharer_sharetime=1600403164443&sharer_shareid=407c90840c4caeeaf9680b1dd38c62ba&key=2ff6efa5b41f0127c9e924be5ce6384145d0c7b612a7633330d08c5fa39fbbfb2ab6c7d4b937423b54f24df15180312494985a7d0d2daa7a62627643cc000fa6dd852a5b9448a9f4044d0c688e5376586d1eef888fecca2dadd2ed1ca94685cce2afb8aa23ee2d4b122ab673e1209256add0569914cbc144befe150d14f9956d&ascene=14&uin=MTIwMjI3NTkwNQ==&devicetype=Windows+10+x64&version=62090529&lang=zh_CN&exportkey=AzxByEV7vLvuybMOAm6iNm4=&pass_ticket=pdKql0fF0rGOXvNr/tToA1+AardNoo77GWcTcNS7PpaVOYI2W/vk8qbSO4P5qmER&wx_header=0&winzoom=1">SaltStack</a>，Kubernetes 中的<a target="_blank" rel="noopener" href="https://mp.weixin.qq.com/s?__biz=MzUzNzYxNjAzMg==&mid=2247494380&idx=1&sn=8cac110d7f50d5a3852a8f69796db963&chksm=fae6e123cd9168359b9bba26f708d2b4013cfed0d592dd1d56daece5040b10367f2f51941a41&mpshare=1&scene=24&srcid=09189RwfTmxbBxMcfb2mLjxz&sharer_sharetime=1600403164443&sharer_shareid=407c90840c4caeeaf9680b1dd38c62ba&key=2ff6efa5b41f0127c9e924be5ce6384145d0c7b612a7633330d08c5fa39fbbfb2ab6c7d4b937423b54f24df15180312494985a7d0d2daa7a62627643cc000fa6dd852a5b9448a9f4044d0c688e5376586d1eef888fecca2dadd2ed1ca94685cce2afb8aa23ee2d4b122ab673e1209256add0569914cbc144befe150d14f9956d&ascene=14&uin=MTIwMjI3NTkwNQ==&devicetype=Windows+10+x64&version=62090529&lang=zh_CN&exportkey=AzxByEV7vLvuybMOAm6iNm4=&pass_ticket=pdKql0fF0rGOXvNr/tToA1+AardNoo77GWcTcNS7PpaVOYI2W/vk8qbSO4P5qmER&wx_header=0&winzoom=1">Argo Workflow</a>、<a target="_blank" rel="noopener" href="https://mp.weixin.qq.com/s?__biz=MzUzNzYxNjAzMg==&mid=2247494380&idx=1&sn=8cac110d7f50d5a3852a8f69796db963&chksm=fae6e123cd9168359b9bba26f708d2b4013cfed0d592dd1d56daece5040b10367f2f51941a41&mpshare=1&scene=24&srcid=09189RwfTmxbBxMcfb2mLjxz&sharer_sharetime=1600403164443&sharer_shareid=407c90840c4caeeaf9680b1dd38c62ba&key=2ff6efa5b41f0127c9e924be5ce6384145d0c7b612a7633330d08c5fa39fbbfb2ab6c7d4b937423b54f24df15180312494985a7d0d2daa7a62627643cc000fa6dd852a5b9448a9f4044d0c688e5376586d1eef888fecca2dadd2ed1ca94685cce2afb8aa23ee2d4b122ab673e1209256add0569914cbc144befe150d14f9956d&ascene=14&uin=MTIwMjI3NTkwNQ==&devicetype=Windows+10+x64&version=62090529&lang=zh_CN&exportkey=AzxByEV7vLvuybMOAm6iNm4=&pass_ticket=pdKql0fF0rGOXvNr/tToA1+AardNoo77GWcTcNS7PpaVOYI2W/vk8qbSO4P5qmER&wx_header=0&winzoom=1">clusterloader2</a>。它们的思想整体比较类似，将高频使用的操作封装为原语，使用者通过原语来表述操作逻辑。</p>
<p>通过声明式的方法，将面向 K8s 的操作抽象成 yaml 中的关键词，在 yaml 中提供串行、并行等控制逻辑，那么就可以通过 yaml 文件完整描述想要进行的工作。</p>
<p>这种思想和<a target="_blank" rel="noopener" href="https://mp.weixin.qq.com/s?__biz=MzUzNzYxNjAzMg==&mid=2247494380&idx=1&sn=8cac110d7f50d5a3852a8f69796db963&chksm=fae6e123cd9168359b9bba26f708d2b4013cfed0d592dd1d56daece5040b10367f2f51941a41&mpshare=1&scene=24&srcid=09189RwfTmxbBxMcfb2mLjxz&sharer_sharetime=1600403164443&sharer_shareid=407c90840c4caeeaf9680b1dd38c62ba&key=2ff6efa5b41f0127c9e924be5ce6384145d0c7b612a7633330d08c5fa39fbbfb2ab6c7d4b937423b54f24df15180312494985a7d0d2daa7a62627643cc000fa6dd852a5b9448a9f4044d0c688e5376586d1eef888fecca2dadd2ed1ca94685cce2afb8aa23ee2d4b122ab673e1209256add0569914cbc144befe150d14f9956d&ascene=14&uin=MTIwMjI3NTkwNQ==&devicetype=Windows+10+x64&version=62090529&lang=zh_CN&exportkey=AzxByEV7vLvuybMOAm6iNm4=&pass_ticket=pdKql0fF0rGOXvNr/tToA1+AardNoo77GWcTcNS7PpaVOYI2W/vk8qbSO4P5qmER&wx_header=0&winzoom=1">Argo Workflow</a> 比较像，但粒度比 Argo 更细，关注在操作函数上：</p>
<p><img src="http://iubest.gitee.io/pic/640-1601016578136.webp" alt="img"></p>
<p>下面简单描述该服务的设计和实现。</p>
<p><strong>设计和实现</strong></p>
<h3 id="1-服务形态"><a href="#1-服务形态" class="headerlink" title="1. 服务形态"></a><strong>1. 服务形态</strong></h3><ul>
<li>使用者在 yaml 中，通过 <strong>声明式</strong> 的方式描述操作逻辑；</li>
<li>以 all-in-one 的二进制工具或 Operator 的方式交付；</li>
<li>服务内置常见原语的实现，以关键字的方式在 yaml 中提供；</li>
<li>支持配置原生 K8s 资源。</li>
</ul>
<h3 id="-1"><a href="#-1" class="headerlink" title=""></a></h3><h3 id="2-设计"><a href="#2-设计" class="headerlink" title="2. 设计"></a><strong>2. 设计</strong></h3><p>该方案的核心在于配置管理的设计，将操作流程配置化，自上而下有如下概念：</p>
<ul>
<li><p><strong>Service</strong>：Modules 或 Tasks 的编排；</p>
</li>
<li><p><strong>Module</strong>：一种任务场景，是操作单元的集合（其中包含 templates/ 目录，表征模板文件的集合，可用来配置 K8s 原生资源）；</p>
</li>
<li><p><strong>Task</strong>：操作单元，使用 plugin 及参数执行操作；</p>
</li>
<li><p><strong>Plugin</strong>：操作指令，类似开发语言中的函数。</p>
</li>
</ul>
<p>抽象目标场景中的通用操作，这些通用操作即为可在 yaml 中使用的原语，对应上述 Plugin：</p>
<ul>
<li><p><strong>K8s 相关</strong></p>
</li>
<li><ul>
<li>CreateNamespace</li>
<li>DeleteNamespace</li>
<li>PrepareSecret</li>
<li>PrepareConfigMap</li>
<li>PrepareBatchDeployments</li>
<li>WaitForBatchDeploymentsReady</li>
<li>etc.</li>
</ul>
</li>
<li><p><strong>观测性相关</strong></p>
</li>
<li><ul>
<li>DeploymentCreationEfficiency</li>
<li>PodCreationEfficiency</li>
<li>etc.</li>
</ul>
</li>
<li><p><strong>检测项相关</strong></p>
</li>
<li><ul>
<li>CheckPodAnnotations</li>
<li>CheckPodObjectInfo</li>
<li>CheckPodInnerStates</li>
<li>etc.</li>
</ul>
</li>
<li><p><strong>控制语句相关</strong></p>
</li>
<li><ul>
<li>RepeatNTimes</li>
<li>etc.</li>
</ul>
</li>
</ul>
<p>上述 4 个概念的关系如下：</p>
<p><img src="http://iubest.gitee.io/pic/640-1601016578094.webp" alt="img"></p>
<p>示例可参见文章开头的 yaml 文件，对应形式二。</p>
<h3 id="-2"><a href="#-2" class="headerlink" title=""></a></h3><h3 id="3-核心实现"><a href="#3-核心实现" class="headerlink" title="3. 核心实现"></a><strong>3. 核心实现</strong></h3><p>CRD 设计：</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br></pre></td><td class="code"><pre><span class="line"></span><br><span class="line">package v1alpha1</span><br><span class="line"></span><br><span class="line">import (</span><br><span class="line">  corev1 &quot;k8s.io&#x2F;api&#x2F;core&#x2F;v1&quot;</span><br><span class="line">  metav1 &quot;k8s.io&#x2F;apimachinery&#x2F;pkg&#x2F;apis&#x2F;meta&#x2F;v1&quot;</span><br><span class="line">)</span><br><span class="line"></span><br><span class="line">&#x2F;&#x2F; BeidouType is the type related to Beidou execution.</span><br><span class="line">type BeidouType string</span><br><span class="line"></span><br><span class="line">const (</span><br><span class="line">  &#x2F;&#x2F; BeidouTask represents the Task execution type.</span><br><span class="line">  BeidouTask BeidouType &#x3D; &quot;Task&quot;</span><br><span class="line">)</span><br><span class="line"></span><br><span class="line">&#x2F;&#x2F; +genclient</span><br><span class="line">&#x2F;&#x2F; +k8s:deepcopy-gen:interfaces&#x3D;k8s.io&#x2F;apimachinery&#x2F;pkg&#x2F;runtime.Object</span><br><span class="line"></span><br><span class="line">&#x2F;&#x2F; Beidou represents a crd used to describe serices.</span><br><span class="line">type Beidou struct &#123;</span><br><span class="line">  metav1.TypeMeta   &#96;json:&quot;,inline&quot;&#96;</span><br><span class="line">  metav1.ObjectMeta &#96;json:&quot;metadata,omitempty&quot; protobuf:&quot;bytes,1,opt,name&#x3D;metadata&quot;&#96;</span><br><span class="line"></span><br><span class="line">  Spec   BeidouSpec   &#96;json:&quot;spec,omitempty&quot; protobuf:&quot;bytes,2,opt,name&#x3D;spec&quot;&#96;</span><br><span class="line">  Status BeidouStatus &#96;json:&quot;status,omitempty&quot; protobuf:&quot;bytes,3,opt,name&#x3D;status&quot;&#96;</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line">&#x2F;&#x2F; BeidouSpec is the spec of a Beidou.</span><br><span class="line">type BeidouSpec struct &#123;</span><br><span class="line">  Steps      []BeidouStep      &#96;json:&quot;steps&quot; protobuf:&quot;bytes,1,opt,name&#x3D;steps&quot;&#96;</span><br><span class="line">  References []BeidouReference &#96;json:&quot;references&quot; protobuf:&quot;bytes,2,opt,name&#x3D;references&quot;&#96;</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line">&#x2F;&#x2F; BeidouStep is the spec of step.</span><br><span class="line">type BeidouStep struct &#123;</span><br><span class="line">  Name       string            &#96;json:&quot;name&quot; protobuf:&quot;bytes,1,opt,name&#x3D;name&quot;&#96;</span><br><span class="line">  Operations []BeidouOperation &#96;json:&quot;operations&quot; protobuf:&quot;bytes,2,opt,name&#x3D;operations&quot;&#96;</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line">&#x2F;&#x2F; BeidouOperation is the spec of operation.</span><br><span class="line">type BeidouOperation struct &#123;</span><br><span class="line">  Name string      &#96;json:&quot;name&quot; protobuf:&quot;bytes,1,opt,name&#x3D;name&quot;&#96;</span><br><span class="line">  Type BeidouType  &#96;json:&quot;type&quot; protobuf:&quot;bytes,2,opt,name&#x3D;type&quot;&#96;</span><br><span class="line">  Op   string      &#96;json:&quot;op&quot; protobuf:&quot;bytes,3,opt,name&#x3D;op&quot;&#96;</span><br><span class="line">  Args []BeidouArg &#96;json:&quot;args&quot; protobuf:&quot;bytes,4,opt,name&#x3D;args&quot;&#96;</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line">&#x2F;&#x2F; BeidouArg is the spec of arg.</span><br><span class="line">type BeidouArg struct &#123;</span><br><span class="line">  Name        string                   &#96;json:&quot;name&quot; protobuf:&quot;bytes,1,opt,name&#x3D;name&quot;&#96;</span><br><span class="line">  Value       string                   &#96;json:&quot;value,omitempty&quot; protobuf:&quot;bytes,2,opt,name&#x3D;value&quot;&#96;</span><br><span class="line">  Reference   BeidouOperationReference &#96;json:&quot;reference,omitempty&quot; protobuf:&quot;bytes,3,opt,name&#x3D;reference&quot;&#96;</span><br><span class="line">  Tolerations []corev1.Toleration      &#96;json:&quot;tolerations,omitempty&quot; protobuf:&quot;bytes,4,opt,name&#x3D;tolerations&quot;&#96;</span><br><span class="line">  Checking    []string                 &#96;json:&quot;checking,omitempty&quot; protobuf:&quot;bytes,5,opt,name&#x3D;checking&quot;&#96;</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line">&#x2F;&#x2F; BeidouOperationReference is the spec of operation reference.</span><br><span class="line">type BeidouOperationReference struct &#123;</span><br><span class="line">  ID string &#96;json:&quot;id&quot; protobuf:&quot;bytes,1,opt,name&#x3D;id&quot;&#96;</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line">&#x2F;&#x2F; BeidouReference is the spec of reference.</span><br><span class="line">type BeidouReference struct &#123;</span><br><span class="line">  ID    string       &#96;json:&quot;id&quot; protobuf:&quot;bytes,1,opt,name&#x3D;id&quot;&#96;</span><br><span class="line">  Steps []BeidouStep &#96;json:&quot;steps&quot; protobuf:&quot;bytes,2,opt,name&#x3D;steps&quot;&#96;</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line">&#x2F;&#x2F; BeidouStatus represents the current state of a Beidou.</span><br><span class="line">type BeidouStatus struct &#123;</span><br><span class="line">  Message string &#96;json:&quot;message&quot; protobuf:&quot;bytes,1,opt,name&#x3D;message&quot;&#96;</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line">&#x2F;&#x2F; +k8s:deepcopy-gen:interfaces&#x3D;k8s.io&#x2F;apimachinery&#x2F;pkg&#x2F;runtime.Object</span><br><span class="line"></span><br><span class="line">&#x2F;&#x2F; BeidouList is a collection of Beidou.</span><br><span class="line">type BeidouList struct &#123;</span><br><span class="line">  metav1.TypeMeta &#96;json:&quot;,inline&quot;&#96;</span><br><span class="line">  metav1.ListMeta &#96;json:&quot;metadata&quot; protobuf:&quot;bytes,1,opt,name&#x3D;metadata&quot;&#96;</span><br><span class="line"></span><br><span class="line">  Items []Beidou &#96;json:&quot;items&quot; protobuf:&quot;bytes,2,opt,name&#x3D;items&quot;&#96;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure>

<p>核心流程：</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br></pre></td><td class="code"><pre><span class="line">&#x2F;&#x2F; ExecSteps executes steps.</span><br><span class="line">func ExecSteps(ctx context.Context, steps []v1alpha1.BeidouStep, references []v1alpha1.BeidouReference) error &#123;</span><br><span class="line">    logger, _ :&#x3D; ctx.Value(CtxLogger).(*log.Entry)</span><br><span class="line"></span><br><span class="line">  var hasMonitored bool</span><br><span class="line">  for i, step :&#x3D; range steps &#123;</span><br><span class="line">    for j, op :&#x3D; range step.Operations &#123;</span><br><span class="line">      switch op.Op &#123;</span><br><span class="line">      case &quot;DeploymentCreationEfficiency&quot;:</span><br><span class="line">        if !hasMonitored &#123;</span><br><span class="line">          defer func() &#123;</span><br><span class="line">            err :&#x3D; monitor.Output()</span><br><span class="line">            if err !&#x3D; nil &#123;</span><br><span class="line">              logger.Errorf(&quot;Failed to output: %s&quot;, err)</span><br><span class="line">            &#125;</span><br><span class="line">          &#125;()</span><br><span class="line">        &#125;</span><br><span class="line">        hasMonitored &#x3D; true</span><br><span class="line">      &#125;</span><br><span class="line"></span><br><span class="line">      err :&#x3D; ExecOperation(ctx, op, references)</span><br><span class="line">      if err !&#x3D; nil &#123;</span><br><span class="line">        return fmt.Errorf(&quot;failed to run operation %s: %s&quot;, op.Name, err)</span><br><span class="line">      &#125;</span><br><span class="line">    &#125;</span><br><span class="line">  &#125;</span><br><span class="line"></span><br><span class="line">  return nil</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line">&#x2F;&#x2F; ExecOperation executes operation.</span><br><span class="line">func ExecOperation(ctx context.Context, op v1alpha1.BeidouOperation, references []v1alpha1.BeidouReference) error &#123;</span><br><span class="line">  switch op.Type &#123;</span><br><span class="line">  case v1alpha1.BeidouTask:</span><br><span class="line">    if !tasks.IsRegistered(op.Op) &#123;</span><br><span class="line">      return ErrNotRegistered</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line">    if !tasks.DoesSupportReference(op.Op) &#123;</span><br><span class="line">      return ExecTask(ctx, op.Op, op.Args)</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line">    return ExecTaskWithRefer(ctx, op.Op, op.Args, references)</span><br><span class="line">  &#125;</span><br><span class="line"></span><br><span class="line">  return nil</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line">&#x2F;&#x2F; ExecTask executes a task.</span><br><span class="line">func ExecTask(ctx context.Context, opname string, args []v1alpha1.BeidouArg) error &#123;</span><br><span class="line">  switch opname &#123;</span><br><span class="line">  case tasks.CreateNamespace:</span><br><span class="line">    var ns string</span><br><span class="line">    for _, arg :&#x3D; range args &#123;</span><br><span class="line">      switch arg.Name &#123;</span><br><span class="line">      case &quot;NS&quot;:</span><br><span class="line">        ns &#x3D; arg.Value</span><br><span class="line">      &#125;</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line">    return op.CreateNamespace(ctx, ns)</span><br><span class="line">    &#x2F;&#x2F; ...</span><br><span class="line">    &#125;</span><br><span class="line">    &#x2F;&#x2F; ...</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line">&#x2F;&#x2F; ExecTaskWithRefer executes a task with reference.</span><br><span class="line">func ExecTaskWithRefer(ctx context.Context, opname string, args []v1alpha1.BeidouArg, references []v1alpha1.BeidouReference) error &#123;</span><br><span class="line">  switch opname &#123;</span><br><span class="line">  case tasks.RepeatNTimes:</span><br><span class="line">    var times int</span><br><span class="line">    var steps []v1alpha1.BeidouStep</span><br><span class="line">    var err error</span><br><span class="line">    for _, arg :&#x3D; range args &#123;</span><br><span class="line">      switch arg.Name &#123;</span><br><span class="line">      case &quot;TIMES&quot;:</span><br><span class="line">        times, err &#x3D; strconv.Atoi(arg.Value)</span><br><span class="line">        if err !&#x3D; nil &#123;</span><br><span class="line">          return ErrParseArgs</span><br><span class="line">        &#125;</span><br><span class="line">      case &quot;ACTION&quot;:</span><br><span class="line">        for _, refer :&#x3D; range references &#123;</span><br><span class="line">          if refer.ID &#x3D;&#x3D; arg.Reference.ID &#123;</span><br><span class="line">            steps &#x3D; refer.Steps</span><br><span class="line">            break</span><br><span class="line">          &#125;</span><br><span class="line">        &#125;</span><br><span class="line">      &#125;</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line">    return RepeatNTimes(ctx, times, steps)</span><br><span class="line">  &#125;</span><br><span class="line"></span><br><span class="line">  return ErrNotImplemented</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure>

<p>操作原语的实现示例：</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br></pre></td><td class="code"><pre><span class="line">&#x2F;&#x2F; PodAnnotations is an operation used to check whether annotations of Pod are expected.</span><br><span class="line">func PodAnnotations(ctx context.Context, data PodAnnotationsData) error &#123;</span><br><span class="line">  kclient, ok :&#x3D; ctx.Value(tasks.KubernetesClient).(kubernetes.Interface)</span><br><span class="line">  if !ok &#123;</span><br><span class="line">    return tasks.ErrNoKubernetesClient</span><br><span class="line">  &#125;</span><br><span class="line"></span><br><span class="line">  pods, err :&#x3D; kclient.CoreV1().Pods(data.Namespace).List(metav1.ListOptions&#123;&#125;)</span><br><span class="line">  if err !&#x3D; nil &#123;</span><br><span class="line">    return fmt.Errorf(&quot;failed to list pods in ns %s: %s&quot;, data.Namespace, err)</span><br><span class="line">  &#125;</span><br><span class="line"></span><br><span class="line">  for _, pod :&#x3D; range pods.Items &#123;</span><br><span class="line">    if pod.Annotations &#x3D;&#x3D; nil &#123;</span><br><span class="line">      return fmt.Errorf(&quot;pod %s in ns %s has no annotations&quot;, pod.Name, data.Namespace)</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line">    for _, annotation :&#x3D; range data.Exists &#123;</span><br><span class="line">      if _, exists :&#x3D; pod.Annotations[annotation]; !exists &#123;</span><br><span class="line">        return fmt.Errorf(&quot;annotation %s does not exist in pod %s in ns %s&quot;, annotation, pod.Name, data.Namespace)</span><br><span class="line">      &#125;</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line">    for k, v :&#x3D; range data.Equal &#123;</span><br><span class="line">      if pod.Annotations[k] !&#x3D; v &#123;</span><br><span class="line">        return fmt.Errorf(&quot;value of annotation %s is not %s in pod %s in ns %s&quot;, k, v, pod.Name, data.Namespace)</span><br><span class="line">      &#125;</span><br><span class="line">    &#125;</span><br><span class="line">  &#125;</span><br><span class="line"></span><br><span class="line">  return nil</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure>

<p><strong>后续</strong></p>
<p>目前阿里云容器服务团队内部已经实现了初版，已用于部分云产品的内部性能测试以及常规的回归测试，很大程度上提升了我们的工作效率。</p>
<p>在 yaml 中编程，是对云原生场景下声明式操作的体现，也是对声明式服务的一种实践。对于常规工作场景中重复编码或重复操作，可考虑类似的方式进行满足。</p>
<p>欢迎大家对这样的服务形态和项目进行讨论，探索这种模式的价值。</p>
<p><strong>阿里云容器服务持续招聘，欢迎加入我们，一起在 K8s、边缘计算、Serverless 等领域开拓，让当前变得更美好，也为未来带来可能性！联系邮箱：<a href="mailto:&#x66;&#108;&#x79;&#x65;&#x72;&#x2e;&#x7a;&#x79;&#102;&#64;&#x61;&#108;&#x69;&#x62;&#x61;&#x62;&#x61;&#x2d;&#x69;&#110;&#x63;&#46;&#99;&#111;&#x6d;">&#x66;&#108;&#x79;&#x65;&#x72;&#x2e;&#x7a;&#x79;&#102;&#64;&#x61;&#108;&#x69;&#x62;&#x61;&#x62;&#x61;&#x2d;&#x69;&#110;&#x63;&#46;&#99;&#111;&#x6d;</a></strong></p>
<h3 id="References"><a href="#References" class="headerlink" title="References"></a><strong>References</strong></h3><ul>
<li><strong>Ansible</strong>：<a target="_blank" rel="noopener" href="https://docs.ansible.com/ansible/latest/index.html">https://docs.ansible.com/ansible/latest/index.html</a></li>
<li><strong>SaltStack</strong>：<a target="_blank" rel="noopener" href="https://docs.saltstack.com/en/latest/">https://docs.saltstack.com/en/latest/</a></li>
<li><strong>Argo Workflow</strong>：<a target="_blank" rel="noopener" href="https://github.com/argoproj/argo">https://github.com/argoproj/argo</a></li>
<li><strong>clusterloader2</strong>：<a target="_blank" rel="noopener" href="https://github.com/kubernetes/perf-tests/tree/master/clusterloader2">https://github.com/kubernetes/perf-tests/tree/master/clusterloader2</a></li>
</ul>
 
      <!-- reward -->
      
      <div id="reword-out">
        <div id="reward-btn">
          打赏
        </div>
      </div>
      
    </div>
    

    <!-- copyright -->
    
    <div class="declare">
      <ul class="post-copyright">
        <li>
          <i class="ri-copyright-line"></i>
          <strong>版权声明： </strong>
          
          本博客所有文章除特别声明外，著作权归作者所有。转载请注明出处！
          
        </li>
      </ul>
    </div>
    
    <footer class="article-footer">
       
<div class="share-btn">
      <span class="share-sns share-outer">
        <i class="ri-share-forward-line"></i>
        分享
      </span>
      <div class="share-wrap">
        <i class="arrow"></i>
        <div class="share-icons">
          
          <a class="weibo share-sns" href="javascript:;" data-type="weibo">
            <i class="ri-weibo-fill"></i>
          </a>
          <a class="weixin share-sns wxFab" href="javascript:;" data-type="weixin">
            <i class="ri-wechat-fill"></i>
          </a>
          <a class="qq share-sns" href="javascript:;" data-type="qq">
            <i class="ri-qq-fill"></i>
          </a>
          <a class="douban share-sns" href="javascript:;" data-type="douban">
            <i class="ri-douban-line"></i>
          </a>
          <!-- <a class="qzone share-sns" href="javascript:;" data-type="qzone">
            <i class="icon icon-qzone"></i>
          </a> -->
          
          <a class="facebook share-sns" href="javascript:;" data-type="facebook">
            <i class="ri-facebook-circle-fill"></i>
          </a>
          <a class="twitter share-sns" href="javascript:;" data-type="twitter">
            <i class="ri-twitter-fill"></i>
          </a>
          <a class="google share-sns" href="javascript:;" data-type="google">
            <i class="ri-google-fill"></i>
          </a>
        </div>
      </div>
</div>

<div class="wx-share-modal">
    <a class="modal-close" href="javascript:;"><i class="ri-close-circle-line"></i></a>
    <p>扫一扫，分享到微信</p>
    <div class="wx-qrcode">
      <img src="//api.qrserver.com/v1/create-qr-code/?size=150x150&data=http://example.com/2020/11/11/k8s/Kubernetes%20%E6%96%B0%E7%8E%A9%E6%B3%95%EF%BC%9A%E5%9C%A8%20yaml%20%E4%B8%AD%E7%BC%96%E7%A8%8B/" alt="微信分享二维码">
    </div>
</div>

<div id="share-mask"></div>  
  <ul class="article-tag-list" itemprop="keywords"><li class="article-tag-list-item"><a class="article-tag-list-link" href="/tags/k8s/" rel="tag">k8s</a></li></ul>

    </footer>
  </div>

   
  <nav class="article-nav">
    
      <a href="/2020/11/11/k8s/K8S%E4%BD%BF%E7%94%A8ceph-csi%E6%8C%81%E4%B9%85%E5%8C%96%E5%AD%98%E5%82%A8%E4%B9%8BCephFS/" class="article-nav-link">
        <strong class="article-nav-caption">上一篇</strong>
        <div class="article-nav-title">
          
            K8S使用ceph-csi持久化存储之CephFS.md
          
        </div>
      </a>
    
    
      <a href="/2020/11/11/k8s/kubernetes%20%E5%B8%B8%E8%A7%81%E9%97%AE%E9%A2%98%E6%80%BB%E7%BB%93/" class="article-nav-link">
        <strong class="article-nav-caption">下一篇</strong>
        <div class="article-nav-title">kubernetes 常见问题总结.md</div>
      </a>
    
  </nav>

   
<!-- valine评论 -->
<div id="vcomments-box">
  <div id="vcomments"></div>
</div>
<script src="//cdn1.lncld.net/static/js/3.0.4/av-min.js"></script>
<script src="https://cdn.jsdelivr.net/npm/valine@1.4.14/dist/Valine.min.js"></script>
<script>
  new Valine({
    el: "#vcomments",
    app_id: "",
    app_key: "",
    path: window.location.pathname,
    avatar: "monsterid",
    placeholder: "给我的文章加点评论吧~",
    recordIP: true,
  });
  const infoEle = document.querySelector("#vcomments .info");
  if (infoEle && infoEle.childNodes && infoEle.childNodes.length > 0) {
    infoEle.childNodes.forEach(function (item) {
      item.parentNode.removeChild(item);
    });
  }
</script>
<style>
  #vcomments-box {
    padding: 5px 30px;
  }

  @media screen and (max-width: 800px) {
    #vcomments-box {
      padding: 5px 0px;
    }
  }

  #vcomments-box #vcomments {
    background-color: #fff;
  }

  .v .vlist .vcard .vh {
    padding-right: 20px;
  }

  .v .vlist .vcard {
    padding-left: 10px;
  }
</style>

 
     
</article>

</section>
      <footer class="footer">
  <div class="outer">
    <ul>
      <li>
        Copyrights &copy;
        2015-2020
        <i class="ri-heart-fill heart_icon"></i> TzWind
      </li>
    </ul>
    <ul>
      <li>
        
        
        
        由 <a href="https://hexo.io" target="_blank">Hexo</a> 强力驱动
        <span class="division">|</span>
        主题 - <a href="https://github.com/Shen-Yu/hexo-theme-ayer" target="_blank">Ayer</a>
        
      </li>
    </ul>
    <ul>
      <li>
        
        
        <span>
  <span><i class="ri-user-3-fill"></i>访问人数:<span id="busuanzi_value_site_uv"></span></s>
  <span class="division">|</span>
  <span><i class="ri-eye-fill"></i>浏览次数:<span id="busuanzi_value_page_pv"></span></span>
</span>
        
      </li>
    </ul>
    <ul>
      
    </ul>
    <ul>
      
    </ul>
    <ul>
      <li>
        <!-- cnzz统计 -->
        
        <script type="text/javascript" src='https://s9.cnzz.com/z_stat.php?id=1278069914&amp;web_id=1278069914'></script>
        
      </li>
    </ul>
  </div>
</footer>
      <div class="float_btns">
        <div class="totop" id="totop">
  <i class="ri-arrow-up-line"></i>
</div>

<div class="todark" id="todark">
  <i class="ri-moon-line"></i>
</div>

      </div>
    </main>
    <aside class="sidebar on">
      <button class="navbar-toggle"></button>
<nav class="navbar">
  
  <div class="logo">
    <a href="/"><img src="/images/ayer-side.svg" alt="Hexo"></a>
  </div>
  
  <ul class="nav nav-main">
    
    <li class="nav-item">
      <a class="nav-item-link" href="/">主页</a>
    </li>
    
    <li class="nav-item">
      <a class="nav-item-link" href="/archives">归档</a>
    </li>
    
    <li class="nav-item">
      <a class="nav-item-link" href="/categories">分类</a>
    </li>
    
    <li class="nav-item">
      <a class="nav-item-link" href="/tags">标签</a>
    </li>
    
    <li class="nav-item">
      <a class="nav-item-link" target="_blank" rel="noopener" href="http://www.baidu.com">百度</a>
    </li>
    
    <li class="nav-item">
      <a class="nav-item-link" href="/friends">友链</a>
    </li>
    
    <li class="nav-item">
      <a class="nav-item-link" href="/2019/about">关于我</a>
    </li>
    
  </ul>
</nav>
<nav class="navbar navbar-bottom">
  <ul class="nav">
    <li class="nav-item">
      
      <a class="nav-item-link nav-item-search"  title="搜索">
        <i class="ri-search-line"></i>
      </a>
      
      
      <a class="nav-item-link" target="_blank" href="/atom.xml" title="RSS Feed">
        <i class="ri-rss-line"></i>
      </a>
      
    </li>
  </ul>
</nav>
<div class="search-form-wrap">
  <div class="local-search local-search-plugin">
  <input type="search" id="local-search-input" class="local-search-input" placeholder="Search...">
  <div id="local-search-result" class="local-search-result"></div>
</div>
</div>
    </aside>
    <script>
      if (window.matchMedia("(max-width: 768px)").matches) {
        document.querySelector('.content').classList.remove('on');
        document.querySelector('.sidebar').classList.remove('on');
      }
    </script>
    <div id="mask"></div>

<!-- #reward -->
<div id="reward">
  <span class="close"><i class="ri-close-line"></i></span>
  <p class="reward-p"><i class="ri-cup-line"></i>请我喝杯咖啡吧~</p>
  <div class="reward-box">
    
    
  </div>
</div>
    
<script src="/js/jquery-2.0.3.min.js"></script>


<script src="/js/lazyload.min.js"></script>

<!-- Tocbot -->


<script src="/js/tocbot.min.js"></script>

<script>
  tocbot.init({
    tocSelector: '.tocbot',
    contentSelector: '.article-entry',
    headingSelector: 'h1, h2, h3, h4, h5, h6',
    hasInnerContainers: true,
    scrollSmooth: true,
    scrollContainer: 'main',
    positionFixedSelector: '.tocbot',
    positionFixedClass: 'is-position-fixed',
    fixedSidebarOffset: 'auto'
  });
</script>

<script src="https://cdn.jsdelivr.net/npm/jquery-modal@0.9.2/jquery.modal.min.js"></script>
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/jquery-modal@0.9.2/jquery.modal.min.css">
<script src="https://cdn.jsdelivr.net/npm/justifiedGallery@3.7.0/dist/js/jquery.justifiedGallery.min.js"></script>

<script src="/dist/main.js"></script>

<!-- ImageViewer -->

<!-- Root element of PhotoSwipe. Must have class pswp. -->
<div class="pswp" tabindex="-1" role="dialog" aria-hidden="true">

    <!-- Background of PhotoSwipe. 
         It's a separate element as animating opacity is faster than rgba(). -->
    <div class="pswp__bg"></div>

    <!-- Slides wrapper with overflow:hidden. -->
    <div class="pswp__scroll-wrap">

        <!-- Container that holds slides. 
            PhotoSwipe keeps only 3 of them in the DOM to save memory.
            Don't modify these 3 pswp__item elements, data is added later on. -->
        <div class="pswp__container">
            <div class="pswp__item"></div>
            <div class="pswp__item"></div>
            <div class="pswp__item"></div>
        </div>

        <!-- Default (PhotoSwipeUI_Default) interface on top of sliding area. Can be changed. -->
        <div class="pswp__ui pswp__ui--hidden">

            <div class="pswp__top-bar">

                <!--  Controls are self-explanatory. Order can be changed. -->

                <div class="pswp__counter"></div>

                <button class="pswp__button pswp__button--close" title="Close (Esc)"></button>

                <button class="pswp__button pswp__button--share" style="display:none" title="Share"></button>

                <button class="pswp__button pswp__button--fs" title="Toggle fullscreen"></button>

                <button class="pswp__button pswp__button--zoom" title="Zoom in/out"></button>

                <!-- Preloader demo http://codepen.io/dimsemenov/pen/yyBWoR -->
                <!-- element will get class pswp__preloader--active when preloader is running -->
                <div class="pswp__preloader">
                    <div class="pswp__preloader__icn">
                        <div class="pswp__preloader__cut">
                            <div class="pswp__preloader__donut"></div>
                        </div>
                    </div>
                </div>
            </div>

            <div class="pswp__share-modal pswp__share-modal--hidden pswp__single-tap">
                <div class="pswp__share-tooltip"></div>
            </div>

            <button class="pswp__button pswp__button--arrow--left" title="Previous (arrow left)">
            </button>

            <button class="pswp__button pswp__button--arrow--right" title="Next (arrow right)">
            </button>

            <div class="pswp__caption">
                <div class="pswp__caption__center"></div>
            </div>

        </div>

    </div>

</div>

<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/photoswipe@4.1.3/dist/photoswipe.min.css">
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/photoswipe@4.1.3/dist/default-skin/default-skin.min.css">
<script src="https://cdn.jsdelivr.net/npm/photoswipe@4.1.3/dist/photoswipe.min.js"></script>
<script src="https://cdn.jsdelivr.net/npm/photoswipe@4.1.3/dist/photoswipe-ui-default.min.js"></script>

<script>
    function viewer_init() {
        let pswpElement = document.querySelectorAll('.pswp')[0];
        let $imgArr = document.querySelectorAll(('.article-entry img:not(.reward-img)'))

        $imgArr.forEach(($em, i) => {
            $em.onclick = () => {
                // slider展开状态
                // todo: 这样不好，后面改成状态
                if (document.querySelector('.left-col.show')) return
                let items = []
                $imgArr.forEach(($em2, i2) => {
                    let img = $em2.getAttribute('data-idx', i2)
                    let src = $em2.getAttribute('data-target') || $em2.getAttribute('src')
                    let title = $em2.getAttribute('alt')
                    // 获得原图尺寸
                    const image = new Image()
                    image.src = src
                    items.push({
                        src: src,
                        w: image.width || $em2.width,
                        h: image.height || $em2.height,
                        title: title
                    })
                })
                var gallery = new PhotoSwipe(pswpElement, PhotoSwipeUI_Default, items, {
                    index: parseInt(i)
                });
                gallery.init()
            }
        })
    }
    viewer_init()
</script>

<!-- MathJax -->

<!-- Katex -->

<!-- busuanzi  -->


<script src="/js/busuanzi-2.3.pure.min.js"></script>


<!-- ClickLove -->

<!-- ClickBoom1 -->

<!-- ClickBoom2 -->

<!-- CodeCopy -->


<link rel="stylesheet" href="/css/clipboard.css">

<script src="https://cdn.jsdelivr.net/npm/clipboard@2/dist/clipboard.min.js"></script>
<script>
  function wait(callback, seconds) {
    var timelag = null;
    timelag = window.setTimeout(callback, seconds);
  }
  !function (e, t, a) {
    var initCopyCode = function(){
      var copyHtml = '';
      copyHtml += '<button class="btn-copy" data-clipboard-snippet="">';
      copyHtml += '<i class="ri-file-copy-2-line"></i><span>COPY</span>';
      copyHtml += '</button>';
      $(".highlight .code pre").before(copyHtml);
      $(".article pre code").before(copyHtml);
      var clipboard = new ClipboardJS('.btn-copy', {
        target: function(trigger) {
          return trigger.nextElementSibling;
        }
      });
      clipboard.on('success', function(e) {
        let $btn = $(e.trigger);
        $btn.addClass('copied');
        let $icon = $($btn.find('i'));
        $icon.removeClass('ri-file-copy-2-line');
        $icon.addClass('ri-checkbox-circle-line');
        let $span = $($btn.find('span'));
        $span[0].innerText = 'COPIED';
        
        wait(function () { // 等待两秒钟后恢复
          $icon.removeClass('ri-checkbox-circle-line');
          $icon.addClass('ri-file-copy-2-line');
          $span[0].innerText = 'COPY';
        }, 2000);
      });
      clipboard.on('error', function(e) {
        e.clearSelection();
        let $btn = $(e.trigger);
        $btn.addClass('copy-failed');
        let $icon = $($btn.find('i'));
        $icon.removeClass('ri-file-copy-2-line');
        $icon.addClass('ri-time-line');
        let $span = $($btn.find('span'));
        $span[0].innerText = 'COPY FAILED';
        
        wait(function () { // 等待两秒钟后恢复
          $icon.removeClass('ri-time-line');
          $icon.addClass('ri-file-copy-2-line');
          $span[0].innerText = 'COPY';
        }, 2000);
      });
    }
    initCopyCode();
  }(window, document);
</script>


<!-- CanvasBackground -->


    
  </div>
</body>

</html>